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INTRODUCTION 


A  job  performance  aid  is  something  that  a  maintenance  man  uses  to 
guide  himself  while  he  is  performii^  a  task  of  his  job.  It  provides  step- 
by-step  instructions  for  the  job  at  hand.  The  most  systematic  explora¬ 
tory  development  work  to  be  found  in  the  research  literature  concerning 
job  peiformance  aids  has  been  conducted  by  the  Air  Force  Human  Resources 
Laboratory  since  1960  and  has  been  concerned  primarily  with  aiding  the 
performance  of  electronic  mainteriance  tasks.  Most  of  the  other  explora¬ 
tory  development  have  likewise  been  concerned  with  electronic  tasks  but 
more  specifically  with  electronic  troubleshooting  tasks.  Although  the 
AFHRL  work  has  considered  many  factors  related  to  job  performance 
aids  area,  its  most  unique  and  far  reachin-g  contribution  has  been  fully 
proceduralized  or  non-decision  aids  for  all  electronic  maintenance 
activities.  The  laboratory  development  and  tryout  ot  these  non-decision 
aids  was  completed  in  1968  with  the  report  by  Elliott,  Currently  the 
AFHRL  is  preparing  fully  proceduralized  aids  for  an  Air  Force  electronics 
subsystem. 

Another  large  Air  Force  Project  that  has  considered  other  factors 
concerning  job  performance  aids  has  been  Air  Force  Project  PIMO 
(Presentation  of  Information  for  Maintenance  and  Operations) ,  The  PIMO 
project  was  started  in  1964  under  the  management  of  the  Ballistic  Systems 
Division  of  the  Air  Force  Systems  Command.  The  project  liad  both  e3q)loratoi7 
and  advanced  development  phases.  The  experiment  has  been  completed 
and  the  final  report  is  being  prepared.  Although  this  was  a  BSD  project, 
AFHRL  was  represented  on  the  working  group  of  this  project  from  its 
start.  In  its  final  form  the  PIMO  project  developed  aids  for  both 
mechanical  and  electronic  tasks.  It  was ,  however,  limited  to  flight  line 
maintenance. 

Most  researchers  have  had  as  their  general  objective  the  improved 
effectiveness  of  systems  maintenance.  Some  have  accepted  current 
selection  and  training  practices  and  have  concentrated  on  improved 
effectiveness  of  maintenance  only.  Others  have  added  such  variables  as 
length  and  content  of  training  and  aptitude  of  trainees.  The  exploratory 
development  efforts  of  the  Training  Research  Division  of  the  Air  Force 
Human  Resources  Laboratory  have  generally  followed  the  latter  approach, 

The  experimental  evidence,  to  date,  indicates  that  great  savings  both 
in  system  maintenance  efficiency  and  in  training  efficiency  can  be  obtained 
by  well  designed  job  performance  aids.  .  If  a  maintenance  man  uses  a  good 
job  performance  aid,  he  reqiures  less  txaining  and  he  makes  fewer  errors 
in  his  work.  But ,  there  seems  to  be  some  confusion  as  to  the  contribution 
that  each  of  the  important  job  performance  aids  efforts  can  make  to  improv¬ 
ing  maintenance  and  reducing  training  time.  The  purpose  of  this  technical 


report  is  to  summarize  the  findings^  beth-the  AFHRL  work  and  the  PIMO 
.  work  as  well  as  other  important-projeGts,  to  compare  the  scope  of  each 
of  the  important  efforts,  to  identify  areas  of  difficulty  in  implementing 
research  findings,  and  to  make  recommendations  for  an  implementation 
program  that  will  obtain  immediate  as  well  as  long  range  gains... 

\ 


AIR  FORCE  HUMAN  REI^URCES  lABORATORY 
EXPLORATORY  DEVELOPMENT  PROGRAM 

Since  1960,  the  Techmcal  Training  Branch  has  supported  a  small  but 
continuing  exploratory  development  program.  From  1960  throi^hFY  88 
this  was  supported  by  a  total  expenditure  of  only  $310K.  This  program  has 
included  (1)  a  systematic  investigation  to  identify  important  variables 
concerning  job  perfor  mance  aids  and  the  study  of  the  effects  of  such 
variables  on  actual  performance  and  (2)  field  survey  to  determine  both 
the  current  and  future  of  Air  Force  Electronic  Maintenance  Technical 
Data  Problems.  For  most  of  this  effort,  since  1962,  the  author  of  this 
summary ,  Dr.  John  P.  Foley,  Jr. ,  of  ^e  A.ir  Force  Human  Resources 
Laboratory,  has  been  the  Air  Force  technical  monitor  of  this  effort.  All 
of  this  program  has  been  conducted  under  contract  and  the  results  have 
been  reported  in  15  technical  reports.  Mr.  Thomas  K.  Elliott  of 
Applied  Science  Associates,  Inc.  (ASA)  of  Valencia  has  been  the  sole  or 
primary  author  of  9  of  these  reports.  Dr.  John  D.  Folley,  Jr. ,  President 
of  Applied  Science  Associates,  Inc. ,  has  been  the  principal  investigator 
for  most  of  this  effort  and  has  been  the  sole  or  primary  author  of  5  of 
these  reports.  Elliott  and  Folley  were  joint  authors  of  only  2  reports. 

Since  1964  most  of  the  actual  v/ork  has  been  performed  by  Thomas  K. 
Elliott. 


AFHRL  Systematic  Investigation 

This  systematic  investigation  has  had  three  phases,  namely  the 
preliminary  developments  and  investigations,  the  study  of  job  performance 
aids  variables  and  the  field  survey. 

1.  The  Preliminary  Developments  and  Investigations  -  This  phase  was 
begun  by  a  1960  review  of  the  job  performance  aids  literature  by  Folley 
and  Hunger.  This  was  followed  by  the  development  and  tryout  of  prelimin¬ 
ary  procedures  for  systematically  designing  job  performance  aids  (Folley, 
1961a  imd  FoUey  and  Shettel,  1962).  These  efforts  pointed  to  the  fact 
that  insufficient  hard  data  existed  for  the  guidance  of  jcb  performance  aid 
developers  and  an  attempt  was  made  to  isolate  problems  requiring  further 
e3q)loratory  developmoit  (Folley,  1961b).  In  addition,  a  maintenance 
task  simulator  was  developed  to  simulate  and  measure  hiunan  activity 
during  the  performance  of  front  panel  checkout  routines  and  during  the 
solving  of  between  stage  troubleshooting  problems.  This  simulator  also 
included  actual  circuits  for  the  study  of  hiunan  behavior  relative  to  solving 
between  stage  troubleshooting  problems.  Three  reports  have  been  published 
describing  this  simulator  -  Elliott  and  Folley,  1964;  Elliott,  1967a  and 
Elliott  1967b. 

1.  Dr.  John  P.  Foley,  Jr.  and  Dr.  John  D.  Folley.  Jr.  are  not  the  same 
person.  Dr.  John  D.  Folley,  Jr.  (middle  initial  D,  and  Folley  spelled  with 
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2.  Study  of  Job  Performance  Aid  Variables  -  A  number  of  reports  have 
resulted,  from  the  investigation  of  variables  such  as  the  following.  (These 
reports  will  be  referenced  later  in  this  summary. ) 

a.  Structure  of  Electronic  Maintenance  Tasks 

(1)  Equipment  checkout  procedures. 

(2)  Align,  adjust  and  calibrate  procedures. 

(3)  Between  ^^e  troubleshooting  procedures. 

(4)  Within  stage  troubleshooting  procedures. 

(5)  Remove  and  replace  activities. 

b.  Stnicture  of  Content  -  The  effectiveness  of  both  decision  and 
non-decision  (fully-proceduraiized)  aids  have  been  studied  and  reported. 

c.  Electronic  Aptitude  -  Subjects  with  both  electronic  aptitude  (75- 
95  percentile)  and  medium  aptitude  (40-60  percentile)  were  used  in  our 
investigations. 

d.  Level  of  Experience  -  Naive  high  school  subjects  as  well  as 
experienced  Air  Force  technicians  were  used  as  subjects. 

e.  Level  of  Detail  -  Block  vs  Schematic  were  studied. 

f.  Presentation  Format  -  Automatic  retrieval  of  visual  information 
vs.  Manual  retrieval  of  visual  information  vs  Automatic  retrieval  of 
audio  information  were  compared. 

3,  Field  Survey  of  Technical  Data  Problems  -  A  field  survey  of  electronic 
maintenance  technical  data  problems  was  made  in  the  1966-67  time  period 
(Folley  and  ElUott,  1967). 


AFHRL  Exploratory  Development  Results 

Results  of  the  AFHRL  Systematic  Investigation  -  The  outcomes  and 
findings  of  our  systematic  investigations  are  summarized  below: 

1.  A  Classification  of  Maintenance  Activities  -  It  is  rather  difficult 
to  determine  the  many  sources  that  contributed  to  the  following  classifica¬ 
tion  of  electronics  maintenance  activities  but  the  author’ s  work  with  this 
AFHRL  esploratory  development,  with  his  work  concerning  job  oriented 
training  and  his  investigation  of  job  performance  measurement  contributed 

two ’*1")  has  been  president  of  Applied  Science  Associates  since  1961.  Prior 
to  19bl  he  was  with  the  American  Institutes  for  Research.  Dr.  John  P.  Foley, 
Jr.  (middle  initial  P  and  Foley  spelled  with  only  one  ”1”)  has  been  an  Air  Force 
employee  since  1942.  He  has  been  an  Air  Force  research  psychologist  since 
1962  at  the  Training  Research  Division  (AFHRL),  Wright-Patterson  AFB,Ohio. 
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to  its  development.  It  first  appeared  in  substantially  this  form  in  1957 
(Foley,  1957).  It  has  proven  to  be  a  very  usefijl  structure  for  the  analysis 
and  organization  of  much  of  our  current  and  proposed  work  in  trainii^-, 
aiding,  and  evaluating  Air  Force  technicians  in  their  job  performance. 

a.  Job  activities  or  tasks  associated  with  electronic  equipment 

(1)  performing  eqvdpment  checkout  procedures 

(2)  adjustii^,  aligning,  and  calibrating 

(3)  isolating  between- st^e  faults  to  particular  stage 

(4)  isolating  within-stage  faults  to  defective  component  I 

(5)  replacing  of  components  I 

b.  Information  gathering  activities  of  electronic  technician  jobs  ' 

,  I- 

(1)  Using  oscilloscope  to:  measure  voltage,  measure  frequency,  f 

compare  waveshape  to  waveshape  standard,  make  high  i 

accuracy  time  base  measurements  and  comparisons  \ 

(2)  Usii^  electronic  voltmeter 

(3)  Using  ohmmeter  ' 

(4)  Using  signal  generator 

(5)  Usii^  tube  checker  to  estimate  quality  of  electron  tubes 

(6)  Using  transistor  checker  to  estimate  quality  of  transistors 

c.  Some  handtools  used  in  electronics  technician  jobs  ; 

(1)  screwdrivers 

(2)  pliers 

(3)  diagonal  cutters 

(4)  soldering  iron 

(5)  solderii^  gun 

(6)  wire  strippers 

(7)  machinist’ s  wrenches 

(8)  light  maciiinist’  s  hammer 
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2.  The  Structui’e  of  Job  Performance  Aid  Content  -  The  effects  of 
both  decision  and  non-decision  aids  routines  on  the  quality  of  job-task 
performance  and  on  training  time  requirements  has  been  studied  ejdensively. 
Subjects  of  various  levels  of  aptitude  and  experience  have  been  included. 

This  study  has  resulted  in  the  following  significant  developments  and 
findings. 


a.  The  development  of  fully  proceduralized  routines  for  between 
stage  and  within  stage  troubleshooting  of  electronic  equipments  has  been 
a  unique  contribution  of  AFHRL  exploratory  development  (Elliott,  1967b). 

b.  High  Aptitude  (70-95)  subjects  perform  between  stage  electronics 
troubleshooting  tasks  better  than  medium  ^0-60)  aptitude  when  using 
decision  type  aids.  These  subjects  were  high  school  students  with 
approxim^ely  e^ht  hours  of  training  who  had  no  previous  electronics 
training  or  experience  (Elliott,  1967a). 

c.  But,  on  the  same  between  stage  troubleshooting  tasks  medium 
aptitude  subjects  perform  as  well  as  high  aptitude  subjects  when  using 
non-decision  orfully-proceduralized  Job  Performance  Aids  (Elliott,  1967a). 

d.  With  only  12  hoxirs  of  training  on  the  use  of  the  fully  proceduralized 
or  non-decision  aids,  on  the  use  of  test  equipment  and  on  the  use  of  hand- 
tools,  both  high  aptitude  and  medium  high  school  subjects  were  able  to 
perform  equipment  checkout  procedures,  between  stage  troubleshooting, 
within  stage  troubleshooting,  and  the  removal  and  replacement  of  parts 
(Elliott,  1967c).  The  electronics  system  used  was  the  Maintenance 

Task  Simulator  (Elliott,  1967d)  which  compares  in  size  and  complexity 
with  a  medium  sized  weapons  control  system. 

e.  Using  the  within  stage  troubleshooting  and  the  removal  and 
replace  portions  of  the  above  tasks,  an  experimental  comparison  was 
made  between  the  performance  of  these  high  school  students  and  the 
performance  of  Air  Force  technicians  averaging  7  years  experience. 

The  high  school  subjects  received  only  12  hours  of  training  but  used 
fully  proceduralized  aids.  The  Air  Force  technicians  used  Technical 
Order  -  like  aids.  These  technicians  had  received  many  months  of 
training.  There  were  no  substantial  differences  in  performance 
between  the  two  groups  (Elliott  and  Joyce,  1968). 

3.  Presentation  Format  -  no  significant  differences  in  effectiveness 
between  the  manual  and  autona  tic  retrieval  of  visual  information,  but 
both  were  superior  to  audio  presentation  of  job  aid  information  (Elliott, 

1966). 
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4.  Level  of  Detail  ~  increased  level  of  detail  interfers  with  both 
speed  and  accuracy  of  electronic  between  stage  troubleshooting  task 
performance.  The  implication  is  that  block  diagram  presentations  of 
informjvtion  flow  data  is  superior  to  schematic  presentation  (Elliott,  1965 
and  Elliott,  1967a). 

5.  Field  Survey  Concerning  Electronic  Maintenance  Technical  Data- 
(Folley  and  Elliott,  1967).  The  followii^  are  but  a  few  of  the  topics 
discussed  in  the  report  of  this  survey. 

a.  Job  Task  Content  -  Many  variations  of  job  structures  are  found 
in  the  Air  Force.  At  one  extreme,  the  technician  is  required  to  identify 
and  replace  faulty  components  to  the  piece-part  level  such  as  resistors  and 
capacitors.  At  the  other  extreme,  the  technician  makes  use  of  semi¬ 
automatic  test  equipment  and  a  fixed  procedure  as  in  the  Minuteman 
System.  If  the  semi-automatic  test  equipment  and  the  procedure  do  not 
solve  the  problem,  it  is  passed  on  to  the  depot  for  correction. 

b.  Troubleshooting  Procedures  -  Apparently  many  techmcians 
frequently  proceed  on  unverified  assumptions,  use  incorrect  logic,  and 
come  to  erroneous  conclusions  on  the  basis  of  imcomplete  evidence. 
Technicians  do  not  keep  adequate  records  of  information  gathered  in  a 
troubleshooting  sequence.  Not  once  was  a  technician  observed  to  have 
written  down  information  from  his  troubleshootii^  checks.  Information 
gathered  early  in  a  troubleshooting  sequence  was  forgotten  before  the  end 
of  the  sequence. 

c.  Retrieval  of  Technical  Order  Information  -  Most  currently 
used  job  performance  aids  are  found  in  technical  orders.  Provisions 
are  generally  inadeq\iatv3  for  retrieval  of  information  by  technicians 
from  Technical  Orders  even  though  most  of  the  required  information  is 
actually  in  the  technical  order.  After  a  series  of  unsuccessful  searches 
for  information  of  a  particular  type,  technicians  tend  to  stop  trying  to 
find  that  type  of  information. 

d.  Performance  Aid;^'  and  Limited  War  -  Nine  technicians  who 
had  recently  returned  from  Sovitheast  Asia  were  interviewed,  and  discus¬ 
sions  were  held  with  responsible  persons  at  Headquarters  Tactical  Air 
Command.  There  appear  to  be  no  unique  problems  with  performance 
aids  in  limited  war  and  counterinsurgency  operations. 

e.  Automatic  Test  Equipment  -  The  increased  use  of  automatic 
or  semi-automatic  test  equipment  has  great  implication  for  simplification 
of  maintenance  jobs.  Such  equipment  genv?  rally  produce  procedural  tasks 
for  the  technician.  The  implications  for  perfcxr  mance  aid  design  and  use 
are  no  different  than  those  for  any  other  procedural  tasks. 
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f.  Miniaturization  -  Miniaturization,  microcircuitry  and  integrated 
circuits  will  probably  reduce  the  importance  of  within-stage  troubleshooting 
to  the  piece  part  level,  even  more.  This  type  of  troubleshooting  may 
largely  disappear. 

g.  Infrared  Pattern  Merprelation  -  The  Air  Force  Aero  Propul¬ 
sion  Laboratory  has  supported  exploratory  development  concerning  the 
infrared  scanning  of  electronic  equipment,  as  a  means  of  equipment 
checkout  and  trouble  isolation  (See  Stahl  and  others,  1967  and 
Vanzetti,  Bobo,  and  Stoddard,  1966).  Such  scannings  result  in  patterns 
for  good  and  defective  equipment.  The  Laboratory  testing  of  this  methodo¬ 
logy  has  been  quite  successful.  To  date  its  application  is  aimed  at 

depot  level  maintenance  since  this  method  requires  controlled  ambient 
temperatures.  The  optimum  presentation  of  infrared  pattern  information 
is  a  performance  aid  problem. 

6.  Design  Handbook  Information  -  As  a  further  result  of  the  AFHRL 
exploratory  development  program,  information  was  furnished  for  the  chapter 
on  Job  Performance  Aids  in  Handbook  of  Instructions  for  Arospace  Personnel 
Subsystem  Design.  (HIAPSD) . 

7.  Civilian  Usage  and  Recognition  -  The  AFHRL  reports  were  refer¬ 
enced  extensively  in  a  U.  S.  Department  of  Labor  Technical  Rq)  ort  con¬ 
cerning  the  impact  of  job  performance  aids  on  manpower  utilization 
(Chalupsky  and  Kopf,  1967). 
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PROJECT  PDVIO 


Description 


PIMO  is  an  acronim  for  Presentation  of  Information  for  Maintenance 
and  Operations.  This  AFSC  project  has  not  been  an  AFHRL  project  al¬ 
though  the  Traming  Research  Division  has  been  represented  on  the 
PIMO  working  group  since  it  started  in  1964.  It  has  been  under  the 
management  of  the  Ballistic  Systems  Division  at  Norton  AFB,  California 
(now  the  Space  and  Missile  Systems  Office  -  AF^).  The  project  has 
included  2  early  exploratory  development  efforts  and  an  advanced 
development  operational  e3q)erimento  From  a  money  point  of  view  this 
has  been  a  much  larger  effort  than  the  AFHRL  effort.  To  date,  the 
project  has  cost  $2700K  as  compared  to  $310K  for  the  AFHRL  effort. 

The  PIMO  advanced  development  experiment  as  planned  was  to 
include  the  development  and  operational  evaluation  of  advanced  type  job 
performance  aids  for  both  flight  line  md  field  shop  maintenance  tasks 
for  the  C-141A  aircraft.  Fully  proceduralized  (non-decision)  aids  were 
to  be  developed  for  all  tasks  except  electronic  troubleshooting  tasks  and 
engine  tasks.  Previously  developed  advanced  decision  type  SIMMS  charts 
were  to  be  used  for  electronic  troubleshooting.  The  non-decision  aids 
were  to  be  presented  in  both  a  hardback  or  book  mode  and  rapid  access, 
visual  aid  mode,,  A  large  scale  field  evaluation  was  planned  to  compare 
the  relative  maintenance  effectiveness  of  current  Air  Force  maintenance 
documents,  hardback  PIMO  type  aids,  and  the  audio-visual  PIMO  type 
aids.  Due  to  a  cutback  in  funding,  the  aids  for  the  field  shop  maintenance 
tasks  were  not  developed.  This  experiment  was  originally  designed  for 
obtaining  systems  effectiveness  information  using  gross  effectiveness 
measures,  such  as  down  time  and  tum-around  time  using  only  those 
maintenance  tasks  that  presented  themselves  in  the  operational  flight 
line  environment.  The  design  was  later  modified  to  include  like 
maintenance  tasks  for  all  groups  imder  study. 

Results  of  the  PIMO  Project 

The  PIMO  experiment  has  been  completed  and  the  final  reports  are 
being  completed  (Goff  and  others,  1969).  The  following  is  a  summary 
of  the  results  of  the  experiment  comparisons. 

1.  Essentially  no  difference  ;ln  performance  time  for  experienced 
specialists^  performing  with  PIMO  aids  and  for  those  performing  with 
conventional  T,  O.  aids  and  information. 


1.  The  experienced  specialists  averaged  25  years  of  age  and  approximately 
19. 2  months  e3q)erience  (  5  and  7  skill  level). 
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2.  The  experienced  specialists  made  errors  when  performing  with  T.  O.  ’  s.‘ 
No  errors  were  made  when  these  specialists  used  PIMO  aids. 

3.  The  apprentices^  performii^  with  PIMO  aids  required  slightly  less 
time  on  the  average  to  complete  the  maintenance  tasks  than  the 
e3Q)erienced  specialists  performing  with  PIMO  aids. 

4.  errors  were  made  by  the  apprentices  when  using  PIMO  aids. 

5.  These  same  apprentices  required  approximately  1/3  more  time  to 
perform  the  tasks  using  conventional  T.  O.  aids  and  information  and 
made  more  errors.  The  apprentices  used  the  T.  O.  aids  and 
information  after  they  had  performed  the  tasks  several  times  using  PIMO 
aids.  (Surprisingly,  they  made  fewer  errors  than  the  e3q)erienced 
specialists  using  conventional  T.O,  *  s). 

6.  A  second  grouvi  of  apprentices  were  allowed  to  use  only  conventional 

T.  0. ’ s.  This  group  required  much  more  time  and  made  many  more  errors 
than  either  the  e^erienced  specialists  or  the  apprentices  in  the  comparative 
experiment.  In  two  of  the  three  jobs,  these  apprentices  were  unable  to 
complete  the  jobs. 

7.  A  group  of  individuals  from  unrelated  career  fields  were,  also  given 
the  same  tasks  to  perform  using  PIMO  aids.  Their  performance  was 
similar  to  that  of  the  apprentices  used  in  the  comparative  experiment 
(see  para  3  and  4  above).  They  were  able  to  perform  the  tasks  very 
well  With  no  errors, 

8.  No  significant  difference  was  found  between  audio-visual  and  visual 
(booklet)  presentation  of  data. 


1.,  The  apprentices  averaged  20  years  of  age  and  approximately  5  months 
exposure  to  the  C-141A,  but  they  had  no  previous  experience  on  the  tasks 
used  in  the  experiment. 
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OTHER  IMPORTANT  JOB  PERFORMANCE 
AIDS  PROJECTS 


Decision  Type  Projects 

A  number  of  decision  type  job  performance  aid  systems  have  been 
developed.  Some  of  these  have  been  evaluated  by  well  designed  controlled 
studies  others  have  not.  Shriver  and  Trexler  (1966)  have  described 
and  discussed  several  of  these  systems. 

SIMMS  or  SIMS  (Symbolic  Integrated  Maintenance  Manuals)  -  A 
good  description  of  SIMMS  aids  can  be  found  in  Shriver  and  Trexler 
(1966)  and  therefcre  is  not  repeated  here.  The  SIMMS  aids  have  been 
prepared  for  the  Coast  Guard  and  the  NivJ^y.  The  Air  Force  has  supported 
the  development  of  such  aids  for  three  electronic  systems.  Although 
the  Air  Force  evaluation  of  these  aids  was  rather  indecisive,  it  would 
appear  that  the  SIMMS  cone  ept  adequately  meets  the  needs  for  diagrams 
and  related  information  for  electronic  decision-type  troubleshootii^ 

(Folley  and  Elliott,  1967).  The  Air  Force  has  specifications  available 
for  SIMMS  aids  and  will  prepare  them  for  Major  Command  upon  request. 

The  Navy  has  also  developed  specifications  for  SIMMS  aids. 

HumRRO  Project  FORECAST  -  These  decision  type  aids  are  also 
described  by  Shriver  and  Trexler  (1966).  A  controlled  study  concerning 
the  effectiveness  of  these  aids  indicated  that  training  time  could  be  reduced 
from  30  week  to  12  weeks  for  Army  M-33  Fire  Control  Technicians  (Shriver, 
1960  aiid  Shriver  and  Others,  1964), 


Computer  Generated  Troubleshooting 
Trees 


Both  the  Human  Engineering  Division  of  the  Aerospace  Medical 
Research  Laboratories  and  the  National  Aeronautics  and  Space  Admini¬ 
stration  have  supported  exploratory  development  for  generating  trouble¬ 
shooting  decision  trees  (Folley  and  Pieper,  1964  and  Hannom  and  others, 
1967).  This  work  shows  promise  Zox  the  rapid  generation  of  optimum 
troubleshooting  sequences.  These  sequences  themselves  are  not  in  a 
form  that  is  readily  usable  by  the  technician.  The  sequenced  information 
must  be  transformed  into  either  a  decision  or  non-decision  type  format 
for  presentation.  Such  trees  when  perfected  can  be  utilized  in  the 
development  of  fully  proceduraiized  routines.  The  Human  Engineering 
Division  has  an  active  in-house  project  on  the  computer  generation 
of  troubleshooting  trees. 


II 


Computer  Simulation  of  Circuit  Component 
Variations 


The  Aero  Propulsion  Laboratory  has  supported  exploratory 
development  of  computer  programs  that  simulate  the  effects  of  various 
out-of-tolerance  circuit  component  conditions  on  circuit  output.  The 
computer  can  generate  tables  of  various  out-of-tolerance  output  values 
(symtons)  and  the  probably  component  or  causes  of  each.  This  work  is 
aimed  at  reducing  the  number  of  check  points  necessary  iii  equipment 
for  either  fully  automated  test  equipment  or  tor  proceduralized  trouble¬ 
shooting  routines.  Although  this  method  shows  promise,  a  larg?  amount 
of  expansion  and  refinement  will  be  required  before  this  work  can  be 
made  operational. 


12 


AN  Al-TALYICAL  COMPARLSON  OF  AIDS 
IN  TERMS  OF  MAINTENANCE 
ACTIVITIES  SUPPORTED 

Several  advanced  types  of  joVj  performance  aids  have  been  discussed 
earlier  in  this  summary.  Questions  such  as  the  followii^  are  some¬ 
times  asked  when  new  type  aids  are  mentioned. 

”How  do  these  aids  differ  from  the  data  found  in  current  technical 

orders?'* 

"What*  s  wrong  with  current  technical  order?  The  Air  Force  has 

spent  a  large  amount  of  money  for  complete  data  and  their  adequacy 

has  been  varified  in  the  evaluation  of  the  weapons  system. " 

This  section  has  been  included  to  help  answer  such  questions  and 
to  clarify  what  current  Technical  Orders  and  what  each  of  these  advanced 
type  performance  aids  do  to  assist  or  aid  the  maintenance  man.  Figure  1 
indicates  the  extent  to  which  each  type  of  aid  supports  each  category  or 
function  of  flight  line  or  field  shop  maintenance  activity.  The  categories 
or  functions  of  maintenance  activities  are  the  same  as  those  described 
on  page  5  of  tliis  summary.  Another  set  of  categories  have  been  dev¬ 
eloped  to  describe  various  levels  of  aiding. 

Levels  or  Categories  of  Aiding 

The  levels  of  aiding  were  generated  to  describe  what  each  aid  does 
to  help  the  maintenance  man  perform  a  particular  maintenance  activity. 
These  categories  are  Information  Only  (1),  Restructured  Information  (Rl) , 
Step  by  Step  Procedures  (P)  -  or  directions,  Expanded  Procedures  (EP) 
and  Fully  Proceduralized  (FP)  or  non-decision  aiding.  Perhaps  the  best 
way  to  describe  these  levels  is  to  list  the  attributes  of  fully  proceduralized 
or  non-decision  aids  and  to  indicate  hov/  each  of  the  other  levels  falls 
short  of  these  attributes. 

1.  Fully  Proceduralized  or  Non-Decision  Aids  -  Such  aids  may  appear 
in  various  formats  but  they  can  be  identified  as  "fully-proceduralized"  or 
"non-decision"  aids  if  they  possess  all  of  the  following  attributes. 

a.  Step  by  step  directions  are  provided. 

b.  These  directions  are  based  on  an  action  tree  which  if  followed 
will  demonstrate  to  the  technician  that  the  equipment  under  test  is  either 
operatir^  normally  or  abnormally.  If  any.  abnormality  is  identified,  the 
action  tree  will  provide  a  routine  which  will  insure  that  the  abnormality  is 
identified  and  corrected. 
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r  SIMMS  PIMO  PIMO  AFHRL 


Comparison  of  Performance  Aids  Systems  In  Terms  of  Maintenance  Activities  Supported 


c.  At  every  action  point  in  the  tree,  the  aid  furnishes  of  the 
necessary  information  and  directions  for  the  successful  accomplishment 
of  the  required  action.  The  technician  is  not  required  to  look  for  it  in 
many  parts  of  his  maintenance  document  -  it  is  all  immediately  in  front 
of  him  while  he  is  performing  the  required  action.  These  information 
and  directions  include  (1)  a  picture  or  drawing  of  the  equipment  location 
imder  consideration;  (2)  step  by  step  directions  of  what  he  should  do; 

(3)  specific  information  as  to  what  the  normal  conditions  of  that  location 
should  be  and  (4)  specific  directions  as  to  what  he  should  do  next  if  the 
condition  found  is  either  in  or  out  of  tolerance. 

2.  Procedures  (P)  or  Directions  -  Most  Technical  Orders  provide 
step  by  step  guidance  for  checkout  procedures  and  for  align,  adjust 

and  calibrate  procedures  if  the  technician  can  find  them.  An  examination 
of  this  guidance  will  soon  indicate  that  it  does  not  meet  the  criteria  for 
fully  proceduralized  aids.  He  must  be  familiar  with  the  layout  of  his 
eqxiipment  or  look  for  a  pictorial  in  another  part  of  the  technical  order. 

In  some  cases  tolerances  are  provided  in  another  part  of  the  docmuent 
and  in  others  the  technician  must  make  a  guess  or  judgment  as  to  what 
they  should  be. 

3.  Expanded  Procedures  (EP)  or  Directions  --  Some  of  the  more  advanced 
job  performance  aid  systems  do  contain  an  integrated  action  tree,  for  at  least 
part  of  the  maintenance  actions,  for  example,  the  Maintenance  Dependency 
Charts  of  SIMMS.  These  charts  utilize  much  more  abstract  symbols  than 
the  actual  picture  of  the  location  \mder  consideration.  As  in  the  case  of 
current  technical  orders,  the  technician  must  be  familiar  with  the  physical 
layout  of  his  equipment  or  look  for  a  pictorial  in  another  part  of  the 
document. 

4.  Information  Only  (I)  -  For  some  maintenance  activities  such  as  for 
between  stage  troubleshootii^,  for  within  stage  troubleshooting  and  for 
remove  and  replace  activities,  some  of  the  job  performance  aid  systems 
provide  information.  For  example,  all  technical  maintenance  orders  for 
electronic  equipment  include  a  schematic  diagram  which  provides  a  great 
deal  of  information  in  a  very  abstract  and  symbolic  form  for  both  between 
stage  and  within  stage  troubleshootir^.  The  technician  must,  however, 
develop  his  action  trees  as  he  goes.  He  usually  does  this  each  time  he  is 
confronted  with  an  equipment  which  is  operating  abnormally. 

5.  Restructured  Information  (Rl)  ~  Some  of  the  job  performance  aids  systems 
have  provided  improved  presentations  of  symbolic  and  abstract  infornation. 

For  example,  SIMMS  and  FORECAST  have  blocked  the  schematic  informa¬ 
tion  to  aid  the  technician  in  his  analyses  of  the  schematic  operation. 


Comparison  of  Job  Performance  Aid  Systems 


An  inspection  of  Figure  1  will  indicate  that  each  of  the  systems  is  an 
improvement  over  the  aids  foimd  in  traditional  technical  orders. 

1.  Advanced  Type  Decision  Aids  -  Aids  such  as  SMMS  and  FORECAST 
provide  improved  checkout  procedures.  SIMMS  provides  a  symbolic 
action  tree  which  if  followed  provides  guidance  for  between  stage  and 
systems  troubleshooting.  It  leads  the  technician  to  the  right  ’’bail  park" 
rapidly  provided  he  can  interpret  the  Maintenance  Dependency  Chart. 

Once  he  is  there  he  must  provide  his  own  within  stage  troubleshooting 
procedures  and  locate  his  own  support  information.  Both  SIMMS  and 
FORECAST  provide  blocked  schematics  for  within  stage  troubleshooting. 

2.  Fully  Proceduralized  or  Non-Decision  Aid  Systems  -  Both  the  PIMO 
project  and  the  AFHRE  exploratory  development  efforts  have  produced 
fully  proceduralized  aids.  The  PEMO  project  has  provided  such  aids  for 
checkout  activities;  align,  adjust  and  calibrate  activities;  and  for 
remove  and  replace  activities  for  a  great  many  flight  line  maintenance 
tasks  for  the  C-141  aircraft.  But  for  between  stage  troubleshooting,  it 
has  fallen  back  onto  SIMMS  aids.  This  project  was  limited  to  flight 
line  maintenance  and  did  not  include  the  actual  engine  maintenance  and 
field  shop  maintenance  of  electronic  equipments.  As  a  result,  little 
trouble  analysis  was  required  for  the  spectrum  of  maintenance  tasks  included 
in  the  project.  Between  stage  and  within  stage  trouble  analysis  are  the 
most  difficult  maintenance  tasks  for  maintenance  technicians  to  accomplish 
usii^  conventional  aids.  They  also  have  been  the  most  difficult  activities  for 
which  to  provide  fully  proceduralized  aids.  Our  exploratory  development 
program  has  addressed  itself  to  this  task  of  providing  fully  proceduralized 
procedures  for  between  and  within  stage  troubleshooting  for  electronic 
equipment  and  has  successfully  produced  such  aids. 
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TRAINING  MPUCATIONS 


1 .  When  current  maintenance  jobs  are  completely  restructured  by 
well  designed  job  performance  aids  great  savings  in  training  time  have 
been  demonstrated.  HumRHO  Project  FORECAST  usii^  well  designed 
decision  type  performance  aids  reduced  training  time  from  30  weeks  to 
12  weeks  for  Army  M-33  Fire  Q)ntrol  Technicians.  (Shriver,  1969  and 
Sliriver  and  others,  1964), 

The  AFHRL  research  has  demonstrated  that  high  school  students  of 
both  medium  and  high  electronic  aptitude  can  troubleshoot  after  very  short 
training  periods  using  fully  proceduralized  aids  (Elliott,  1967c).  The 
PIMO  ejqDeriment  has  demonstrated  that  both  maiit  enance  apprentices  and 
personnel  with  no  previous  training  using  fully  proceduralized  aids 
could  perform  C~141  flight  line  maintenance  tasks  as  well  as  the 
experienced  certified  technicians  (Goff,  1969). 

2.  At  the  present  time,  more  immediate  training  savings  can  be 
made  in  some  career  fields  than  in  others.  For  example,  a  study  of 
Table  1  indicates  that  there  are  5691  personnel  authorized  with  AFSC'  s 
C-43r31E  and  C-43151E  for  C-141  flight  line  maintenance.  It  can  be 
estimated  that  on  an  average  approxinately  1000  of  these  people  have  to 

be  trained  each  year.  Their  present  training  requires  16  weeks  of  residence 
training  at  Sheppard  AFB  and  at  least  12  weeks  OJT  or  a  total  of  28  weeks. 
The  results  of  PIMO  indicate  that  if  PIMO  aids  were  applied  to  the  flight 
line  tasks  for  entire  C-141  fleet,  this  training  could  be  accomplished  in  at 
most  4  weeks  or  a  reduction  of  24  weeks.  This  would  add  6  months  of 
effective  usage  of  each  of  these  1000  men.  Based  on  a  cost  of  $25,000 
(LeMay  1964)  per  man  for  a  four  year  enlistment  with  normal  job  utilization 
of  only  one  year,  a  savings  of  at  least  $8000  would  be  made  per  man  or 
a  total  of  $8, 000, 000  per  year  on  only  this  one  AFSC.  Of  course,  applying 
this  concept  to  the  C-135  and  C5A  would  greatly  increase  these  savings. 

3.  The  introduction  of  miniturization,  microcircuitry,  and  integrated 
circuits  will  reduce  the  requirement  for  within  stage  troubleshooting 
schemes.  This  should  shift  training  emphasis  from  knowledge  of  individual 
circuit  operation  and  faulty  piece  part  identification  to  the  practice  of 
solving  between  stage  troubleshooting  problems  (systems  analysis)  and 
practice  in  following  checkout  procedures.  The  technician  will  still  be 
required  to  operate  his  test  equipment  at  a  high  level  of  competence  to 
gather  equipment  information,  to  align  and  adjust  equipment  and  in  some 
cases  to  use  his  handtools  to  replace  defective  equipment  stages  or 
fimctional  imits. 
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PERSONNEL  INFORMATION  CONCERNING  C-141 
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3 
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6 

1 

4 
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6 

3 

5 
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119 

6 

1 

4 

433 

146 

4  c  The  introduction  of  automatic  and  semi-automatic  test  equipment 
will  reduce  both  between  stage  and  within  stage  troubleshooting  tasks  to 
procedurali25ed  tasks  reducing  or  eliminating  the  requirement  for  training 
on  circuit  (stage)  and  systems  analysis. 


CURRENT  AND  FUTURE  AFIIRL  JOB  PERFOR1V5ANCE  AIDS  EFFORTS 

The  Training  Research  Division  of  the  Air  Force  Human  Resources 
I*aboratory  is  developii^  fully  proceduralized  aids  for  the  doppler  radar 
AN/APN-147  and  its  associated  computer  AN/ASN-35  under  contract 
using  $135K  of  exploratory  development  funds.  It  will  be  noted  that  this 
system  is  only  one  of  eight  systems  that  is  the  responsibility  of  AFSC’  s 
30451  and  30471.  Personnel  of  this  career  field  are  responsible  for  both 
the  flight  line  and  field  shop  maintenance.  The  equipment  requires  both 
between  stage  and  within  stage  troubleshooting.  It  is  one  of  the  career 
fields  which  includes  electronic  fundamentals  in  its  training  program. 

The  exploratory  development  program  of  the  AFHRL  in  job  proficiency 
measurement  is  an  independent  effoii  from  this  job  performance  aid  effort 
but  the  job  task  performance  tests  are  being  developed  for  the  AN/APN-147 
and  AN/aSN-35  system.  These  tests  can  therefore  be  used  to  evaluate  the 
effectiveness  of  proposed  job  performance  advanced  development  program. 

The  Training  Research  Division  AFHRL  has  prepared  a  proposed 
advanced  development  plan.  The  proposed  project  is  structured  for  an 
entire  career  field  rather  than  for  an  entire  weapons  system.  This 
would  be  a  four  year  program  to  demonstrate  and  evaluate  the  technology 
for  training  for  electronic  maintenance  jobs  that  have  been  restructuied 
using  both  decision  type  and  fully  proceduralized  job  performance  aids 
is  proposed.  The  effort  will  require  the  development  of  fully  proceduralized 
job  performance  aids  for  an  electronic  career  field;  the  development  of 
applicable  training  programs  to  accompany  both  types  of  aids;  and  the 
accomplishment  of  a  controlled  e^eriment  to  evaluate  the  relative 
maintenance  effectiveness  and  transfer  effectiveness  of  current  procedures, 
decision  type  job  performance  aid  proceaures,  and  fully  proceduralized 
aids  procedures.  The  development  of  the  aids  for  the  AN/APN-i47  -  AH/ 
ASN-35  system  is  but  an  important  preliminary  effort. 

The  evaluations  will  be  made  on  the  basis  of  various  criteria,  such 
as  cost  of  preparation,  timeliness  of  up-dating,  reliability  of  maintenance, 
spare-part  consumption,  manpower  requirements,  training  costs,  and 
transfer  effectiveness  of  training  and  experience.  It  is  estimated  that  this 
advanced  development  program  will  require  $1, 000, 000. 


RECOMMENDATIONS 


In  keeping  with  the  recommendation  of  the  meetii^  of  performance 
aids  researchers  (Folley  and  Elliott,  1967)  our  piimary  efforts  should 
be  aimed  at  getting  the  results  of  previous  exploratory  and  advanced 
developments  utilized  effectively  in  the  operational  Air  Force.  The 
following  actions  are  recommended: 

1.  The  PIMO  aids  developed  for  the  C-141A  aircraft  should  be  up¬ 
dated  and  expanded  to  the  entire  C-141  fleet.  Since  most  of  the  develop¬ 
mental  work  has  been  already  accomplished,  great  improvem^-uts  in  the 
quality  of  flight  line  maintenance  and  reduction  in  personnel  costs  can 
be  realized  with  a  relatively  small  additional  expenditure. 

2.  In  addition,  immediate  attention  should  be  given  to  the  reduction 
or  elimination  of  formal  training  for  personnel  being  trained  for  AFSC- 
C-43131E  which  is  systems  specific  to  the  C-141.  The  training 
curriculum  personnel  no  loiter  required  for  this  training  should  be 
trained  to  develop  and  up-date  the  fully  proceduralized  aids  for  the  C-141. 

3.  The  proposed  advanced  development  program  of  the  Air  Force 
Hiunan  Resources  Laboratory  should  be  accomplished  in  order  that  the 
benefits  of  fully  proceduralized  aids  can  be  applied  to  the  now  most 
difficult  maintenance  activities  -  between  and  within  stage  troubleshooting 
of  electronic  equipment. 

4.  The  procedures  cf  the  PIMO  project  should  be  applied  to  the  flight 
line  maintenance  activities  of  the  C5A.  The  procedures  should  not  be 
applied  to  the  field  shop  maintenance  imtil  the  fully  proceduralized  aids 
being  "polished"  by  the  Air  Force  Human  Resources  Laboratory  are 
available  to  include  all  the  field  shop  maintenance  activities.  This 
includes  between  stage  and  within  stage  troubleshooting.  Study  should 
be  made  of  the  cost  of  applying  these  procedures  to  the  flight  line 
activities  of  the  C-135.  This  would  cover  the  entire  AFSC  431X1E 

in  MAC  with  fully  proceduralized  aids  for  the  flight  line. 

5.  The  Human  Resources  Laboratory  should  initiate  a  study  of  the 
weapons  systems  development  process,  so  that  optimal  job  performance 
aids  trade-offs  can  be  obtained  during  systems  development. 

6.  After  the  above  recommendations  have  been  completed,  the 
Laboratory  should  consider  the  preparation  of  a  proposed  advanced 
development  plan  for  restructuring  the  personnel  classification  and 
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assignment  systems  around  weapons  systems.  K  the  same  type  of  fully 
proceduralized  aids  were  developed  for  all  electromc  subsystems  iind 
mechanical  subsystems,  it  may  be  possible  for  a  person  with  one  AFSC 
to  maintain  a  large  number  of  subsystems.  For  example,  one  type  of 
repairman  could  possibly  be  trained  to  maintain  all  of  the  electromc 
subsystems  of  one  weapons  system  thus  increasing  manpov/er  effectiveness. 
It  is  also  possible  that  such  a  repairman  could  be  trained  in  the  field 
rather  than  in  a  resident  trainii^  center. 

7.  As  an  extension  of  the  study  proposed  in  paragraph  6,  the  progress 
of  fully  automated  checkout  equipment  and  infrared  scanning  devices 
should  be  studied  to  determine  their  possible  impact  on  training,  on 

the  personnel  classification  and  assignment  system,  and  on  manpower 
requirements. 

8.  Micro  electronic  developments  should  be  studied  since  they  too 
may  require  an  advanced  development  program  to  optimize  their  effective¬ 
ness. 
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DIFFICULTIES  ENCOUNTERED  IN  APPLYING 
EXPLORATORY  DEVELOPMENT  FINDINGS 
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1.  The  development  of  advanced  type  aids  currently  cost  more  to  produce 
than  the  traditional  maintenance  data.  Savings  in  both  improved 
systems  effectiveness  or  in  training  time  to  offset  added  production 
cost  have  been  demonstrated,  but  it  may  be  difficult  to  obtain  the 
added  money  needed  for  these  new  type  aids. 

2.  Advanced  development  projects  cost  more  than  e3q)loratory  develop¬ 

ment.  In  an  advanced  development  demonstration  aids  must  be 
developed  for  an  entire  weapons  system  or  an  entire  career  field. 

Even  though  one  modern  aircraft  may  cost  more  than  $1, 000, 000  it 
is  difficult  to  get  money  of  $1,000, 000  magnitude  for  a  project  that 
VDuld  result  in  great  savings  in  operation  of  many  such  aircraft. 

3.  To  demonstrate  systems  effectiveness  an  experiment  must  be  structured 
to  include  an  entire  weapons  system.  But  weapons  systems  are  main¬ 
tained  by  personnel  from  many  career  fields.  In  some  cases  personnel 
from  these  career  fields  maintain  many  other  weapons  systems.  To 
demonstrate  training  savings  an  e:!q)eriment  must  be  structured  to 
include  an  entire  career  field, 

4.  If  the  jobs  of  an  entire  career  field  are  restructured  using  the  same 
type  of  advanced  type  of  aids  and  training  is  structured  around  such  aids, 
maximum  training  savii^s  will  be  effected.  Such  personnel  can  be 
easily  cross-trained  into  career  fields  using  similiar  aids.  Graduates 
must  be  assigned  to  the  specific  jobs  for  v/hich  they  are  trained  or  to 
jobs  similiar ly  restructured.  Therefore,  the  personnel  training  and 
assignment  procedures  must  insure  precise  and  timely  job  assignments. 

5 .  The  content  of  training  for  restructured  lobs  is  considerably  different 
than  traditional  training.  Instructors  would  have  to  learn  how  to  teach 
the  new  materials.  The  instructors  would  have  to  believe  in  the 
effectiveness  of  these  new  materials  (the  performance  aids).  This 
would  require  a  considerable  attitude  change  on  the  part  of  many 
instructors  who  firmly  believe  that  no  one  should  be  allowed  to 
perform  maintenance  without  an  extaisive  exposure  to  electronic  theory. 

6.  A  very  similar  problem  exists  in  the  field  concerning  the  attitudes  of 

maintenance  superintendents  and  supervisors.  Many  also  believe 
that  no  one  should  be  permitted  to  perform  maintenance  without  an 
extensive  knowledge  of  traditional  theory.  In  fact,  some  even  give 
their  new  people  their  own  theory  examination  before  allowing  them 
to  learn  to  maintain  their  equipment.  It  should  be  remembered  that 
current  training  procedures  have  been  in  being  for  over  30  years.  All 
personnel  currently  in  the  field  have  received  such  theory  training. 
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7.  Many  graduates  of  these  programs  probably  could  not  ’’pass”  the 
current  paper  and  pencil  (SKT)  required  for  upgrading  their  AFSC  ’  s. 
They  could,  however,  pass  a  performance  test  on  between  and  within 
stage  troubleshooting  on  their  equipment  and  a  performance  test  on 

the  use  of  test  equipment.  The  upgrading  examinations  for  the  effective 
career  fields  would  have  to  be  greatly  restructured. 

8.  Job-Task  Performance  Test  procedures  together  with  appropriate 

scoring  schemes  must  be  developed  to  measure  both  training  effec¬ 
tiveness  and  job  effectiveness  of  personnel.  Gross  systems  effective¬ 
ness  measures  such  as  down  time  and  turn  around  were  used  in 
a  current  weapons  systems  job  performance  aids  experiment.  Such 
measures  are  not  precise  enough  to  determine  individual  effectiveness. 

9.  Even  though  new  procedures  are  successfully  demonstrated,  there  is 
always  a  resistance  to  change  on  the  part  of  effected  personnel. 
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